Camera system with autonomous miniature camera and light source assembly and method for image enhancement

ABSTRACT

The present Invention relates to a camera system suitable for use in minimally invasive surgery (MIS), among other applications. In at least one embodiment, the camera system includes an autonomous miniature camera, a light source assembly providing features such as steerable illumination and a variable radiation angle, and a control and processing unit for processing images acquired by the camera Io generate improved images having reduced blurring using a deblurring algorithm.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplication no, 61/105,542 entitled “Camera System With AutonomousMiniature Camera And Light Source Assembly And Method For ImageEnhancement Using Color Correlation” filed on Oct. 15, 2008, which ishereby incorporated by reference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT FIELD OFTHE INVENTION

The present invention relates to camera systems and methods and, moreparticularly, to systems and methods of lighting that are employed aspart of (or in combination with) such camera systems and methods in anyof a variety of applications and environments including, for example,medical applications and environments,

BACKGROUND OF THE INVENTION

Minimally invasive surgery (MIS) is a modern surgical technique in whichoperations in the abdomen and thorax or elsewhere are performed throughsmall incisions in the skin. Since 1981, when the first laparoscopiccholecystectomy (gallbladder removal) was performed, this surgical fieldhas greatly improved due to various technological advances. Today thisoperation typically uses 3-6 abdominal skin incisions (eachapproximately 5-12 mm in length), through which optical fibers, cameras,and long operating instruments are inserted into the abdominal cavity.The abdomen is usually insufflated (inflated) with carbon dioxide gas tocreate a working and viewing space and the operation is performed usingthese long instruments through the incisions, images acquired by way ofone or more camera devices inserted into the working and viewing spaceare viewed on a TV monitor beside the patient as the surgery progresses.

One currently available camera for MIS is composed of a 12-15-inch longtube containing several lenses and optical fibers (the laparoscope). Thefront section of the laparoscope enters the abdomen through a portalcalled a “trocar” and the back end is connected to a power source, lightsource and supporting hardware through large cables protruding from therear of the scope. The internal optical fibers convey xenon or halogenlight via an external light source into the internal cavity and thelenses transfer the images from within the cavity to an externallyconnected portable video camera. Disadvantages of this imaging deviceinclude its two-dimensional view, lack of sufficient optical zoom,inability to adjust its angle of view or angle of illumination without anew incision, inability to control light intensity, and restrictions onits movement due to its large size and external cables. This isespecially significant in this era of MIS in which it is desired tominimize the number and extent of incisions, and wherein newtechnologies are being developed such as Natural Orifice TranslumenalEndoscopic Surgery (NOTES) which use no abdominal incisions at all, butinstead utilize a single trocar inserted into the mouth or vagina tocarry all surgical instruments as well as the camera and light source.

Lighting has been a challenge for laparoscopic cameras because theyoperate in an environment of extreme darkness where accessibility tosources of illumination is limited. Further, new cameras are beingutilized which possess features such as a flexible field of view oroptical zoom capability. Although such camera features are extremelydesirable insofar as they allow surgeons to obtain both the global andthe detailed view of the patient's abdomen as well as a large workingspace to practice surgery, it results in more strict requirements onlighting.

More particularly, current light sources including halogen and LED lampsare designed to have a fixed radiation pattern characterized by afull-width half intensity angle. For those cameras for which zoom is anoption, when a surgeon wants to see the details of an organ in a zoom-inmode, the angle of illumination may be much greater than the camera'sfield-of-view and thus there is often insufficient lighting for thedesired image resulting in poor quality of the acquired image. On theother hand, when a surgeon operates the camera in zoom-out mode toachieve a wide field-of-view, the limited illumination angle of thelight source may not cover the entire desired image area, leading torapid fading of the acquired image towards the edge of the image field.In addition, the center portion of the acquired image may be overilluminated and saturated, resulting in penalty in resolution anddynamic range.

For at least the above-described reasons, therefore, it would beadvantageous if an improved system and method for illumination capableof being employed along with cameras as are used in a variety ofapplications including, for example MIS applications, could bedeveloped.

Further, images corresponding to one or more color channels of thecamera can be blurred, and it would be advantageous if an improveddeblurring method could be developed to provide enhanced image quality.

SUMMARY OF THE INVENTION

The present inventors have recognized that such an improved system andmethod for illumination can be provided through the use of one or moretunable lenses as part of the light source. Tunable lenses refer tolenses with variable focal distances, and one type of a tunable lens isa fluidic lens. By utilizing such tunable lenses, which in at least someembodiments are fluidic (or microfluidic) lenses, improved lightingparticularly involving a timed radiation pattern and/or a controlledbeam angle can be achieved.

In at least one embodiment, the present invention relates to a camerasystem suitable for use in minimally invasive surgery (MIS). The camerasystem includes an autonomous miniature camera, a light source assemblyproviding features such as steerable illumination, a variable radiationangle and auto-controlled light intensity, and a control and processingunit for processing images acquired by the camera to generate improvedimages having reduced blurring using a deblurring algorithm whichcorrelates information between colors.

In at least one additional embodiment, the present invention relates toa camera system for MIS having one or more autonomous miniature camerasenabling simultaneous multi-angle viewing and a three dimensional (3D)view, and a light source assembly having a light source with a tunableradiation pattern and dynamic beam steering. The camera is insertableinto an abdominal cavity through the standard portal of the procedure,either through a trocar or via natural orifice. The light sourceassembly is also insertable through the same portal. The camera systemalso preferably includes a control and processing unit for independentlycontrolling the camera (pan, tilt and zoom) and the light sourceassembly. Wireless network links enable communication between thesesystem components and the control and processing unit, eliminating bulkycables. The control and processing unit receives location signals,transmits control signals, and receives acquired image data. The controland processing unit processes the acquired image data to provide highquality images to the surgeon.

In such an embodiment, the tunable and steerable nature of the lightsource assembly allows light to be efficiently focused primarily on thecamera field of view, thereby optimizing the energy directed to theregion of interest. The autonomous nature of the camera and light sourceassembly (in terms of location and movement) allows the camera to befreely movable within the abdominal cavity to allow multiple angles ofview of a region of interest (such as target organs), with the lightsource assembly capable of being adjusted accordingly. In this manner,although camera motion tracking is required, the illumination can beadjusted to reduce glare and provide optimal lighting efficiency.

In still another embodiment of the present invention, the cameraincludes a fluidic lens system which causes at least one color plane(channel) of acquired images to appear sharp and the other color planes(channels) to appear blurred. Therefore, another aspect of the inventionprovides a deblurring algorithm for correcting these blurred colorplanes of acquired images from the camera. The algorithm uses an adaptedperfect reconstruction filter bank that uses high frequency sub-bands ofsham color planes to improve blurred color planes. Refinements can bemade by adding cascades and a filter to the system based on the channelcharacteristics. Blurred color planes have good shading information butpoor edge information. The filter band structure allows the separationof the edge and shading information. During reconstruction, the edgeinformation from the blurred color plane is replaced by the edgeinformation from the sharp color plane.

Further, in at least one embodiment, the present invention relates to acamera system configured to acquire image information representative ofsubject matter within a selected field of view. The camera systemincludes a camera that receives reflected light and based thereongenerates the image information, and a light source assembly operated inconjunction with the camera so as to provide illumination, the lightsource assembly including a first light source and a first tunable lens.At least some of the illumination output by the light source assembly isreceived back at the camera as the reflected light. Additionally, thetunable lens is adjustable to vary the illumination output by the lightsource assembly, whereby the illumination can be varied to substantiallymatch the selected field of view.

Further, in at least one additional embodiment, the present inventionrelates to a method of operating a camera system to acquire imageinformation representative of subject matter within a selected field ofview. The method includes providing a camera and a light sourceassembly, where the light source assembly includes a light source and atunable lens. The method also includes controlling the tunable lens, andtransmitting the light beam from the light source assembly. The methodadditionally includes receiving reflected light at the camera, where atleast some of the light of the light beam transmitted from the lightsource assembly is included as part of the reflected light, and wherethe image information is based at least in part upon the reflectedlight. The tunable lens is controlled to vary the light beam output bythe light source assembly, whereby the light beam can be varied tosubstantially match the selected field of view.

Further, in another aspect of the present invention, a fluidic lenscamera system including an image sensor is used for acquiring image dataof a scene within a field of view of the camera. Image data is providedfrom each of a plurality of color channels of the image sensor, such asred (R), green (G), and blue (B) color channels. A control andprocessing unit is operable to receive and process the acquired imagedata in accordance with image processing methods described below tocorrect the blurred image data. Briefly, wavelet or contourlettransforms can be used to decompose the image data corresponding to thedifferent color channels, mesh the edge information from a sharp colorchannel with the non-edge information from a blurred color channel toform a set of sub-band output coefficients or reconstructioncoefficients in the wavelet or contourlet domain, and then reconstruct adeblurred image using these sub-band output coefficients.

In one embodiment, the decomposition and reconstruction steps use amodified perfect reconstruction filter bank and wavelet transforms. Inanother embodiment, the decomposition and reconstruction steps use acontourlet filter bank, contourlet transforms, and an ant colonyoptimization method for extracting relevant edge information.

In another embodiment, a method for deblurring a blurred first colorimage corresponding to a first color channel of a camera that alsoproduces a sharp second color image corresponding to a second colorchannel of the camera is provided, wherein the first and the secondcolor images each include a plurality of pixels with each pixel havingan associated respective value. The method includes decomposing thefirst color image by filtering and upsampling to generate a first set ofone or more first sub-band output coefficients, wherein each firstsub-band output coefficient corresponds to a respective sub-band in aselected one of a wavelet and a contourlet domain, and decomposing thesecond color image by filtering and upsampling to generate a second setof second sub-band output coefficients, wherein each second sub-handoutput coefficient corresponds to a respective sub-band in the selecteddomain. The method further includes selecting those second sub-bandoutput coefficients that represent edge information, each of theselected second ruby band output coefficients corresponding to arespective selected sub-band, with the selected sub-bands togetherdefining an edge sub-band set, and preparing a third set of sub-bandoutput coefficients which includes the selected second sub-band outputcoefficients representing edge information and at least one firstsub-band output coefficient corresponding to a sub-band other than thosesub-bands in the edge sub-band set. A deblurred first color image isreconstructed by upsampling and filtering using the third set ofsub-band output coefficients as input.

Additionally, in at least some further embodiments, the presentinvention relates to a light source assembly. The light source assemblyincludes an output port, a light source, and a tunable lens positionedbetween the first light source and the output port. The light sourcegenerates light that passes through the tunable lens and then exits thelight source assembly via the output port as output light, and thetunable lens is adjustable to vary a characteristic of the output lightexiting the light source assembly via the output port. In at least someof the above embodiments, the light source includes an array of lightemitting diodes (LEDs) that each can be controlled to be turned on orturned off and, based upon such control, a direction of light exitingthe light source assembly can be varied.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of an exemplary camera system in accordancewith at least one embodiment of the present invention;

FIGS. 2A-2C respectively show three different schematic views of anexemplary light source assembly using an LED and a microfluidic lens,illustrating various tunable radiation patterns achievable by adjustingthe lens;

FIGS. 3A-3C respectively show three different schematic views of anotherembodiment of a light source assembly using LED arrays and amicrofluidic lens, illustrating dynamic beam steering by selection andenergization of one or more LEDs in the array;

FIG. 4 illustrates a perfect reconstruction filter bank for decomposingand reconstructing an image;

FIG. 5 illustrates a modified perfect reconstruction filter bank;

FIG. 6 illustrates a one dimensional perfect reconstruction filter bank;

FIG. 7 illustrates a modified one dimensional perfect reconstructionfilter bank;

FIGS. 8 a-8 b illustrate a contourlet transform and the resultingfrequency division in a contourlet frequency domain;

FIG. 9 illustrates an exemplary two dimensional contourlet filter bank;and

FIGS. 10 a-10 g illustrate a various conditions for a contourlet filterbank.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring to FIG. 1, an exemplary embodiment of a camera system 2includes a miniature camera 4, two light source assemblies 6 and 8,respectively, and a control and processing unit 10 which includes acontrol unit 11 and a PZT (piezoelectric transducer) operator controlunit 12 that are coupled to and in communication with one another. Asshown schematically, the miniature camera 4 and light source assemblies6, 8 are placed within an abdominal cavity 1 (or other body cavity orthe like) of a patient, while the control and processing unit 10 remainsexterior to the body of the patient. In at least one embodiment, thecamera is insertable into an abdominal cavity through a 20 mm incision,such as on the abdominal wall, and the light source assembly is alsoinsertable through an incision. In the present embodiment, it isenvisioned that the miniature camera 4 is physically separated from eachof the light source assemblies 6, 8, each of which is also physicallyseparate from one another. Nevertheless, as represented by a dashed line19, in other embodiments, it is possible for the miniature camera 4 andthe light source assemblies 6, 8 (or, alternatively, one of those lightsource assemblies), to be physically connected or attached, or even tobe housed within the same housing.

The camera system 2 is adjusted such that a field of view 14 of thecamera 4 encompasses a desired region of interest (which in this exampleis a quadrilateral) 16 within the abdominal cavity 1, and thusencompasses subject matter within that desired region of interest (e.g.,a particular organ or portion of an organ). As discussed further below,the light source assemblies 6, 8 are respectively adjusted to generatelight beams 11, 13, respectively, which provide efficient illuminationto the field of view 14 so as to effectively illuminate the desiredregion of interest 16 and subject matter contained therein. Some or allof the light directed to the desired region of interest 16 and generallyto the field of view 14 is reflected off of the subject matter locatedthere and consequently received by the miniature camera 4 as imageinformation. In at least some embodiments, an effort is made to controlthe light provided by the light source assemblies 6, 8 so that theregion illuminated by the light exactly corresponds to (or tills upon),or substantially corresponds to, the field of view 14. Given theadjustability of the light source assemblies in such manner, in at leastsome embodiments the light source assemblies (or light sources) can bereferred to as “smart light source assemblies” or (“smart lightsources”).

As will be described in more detail below with respect to FIGS. 2A-2Cand 3A-3C, in the present embodiment, the miniature camera 4 includes alight sensor and a fluidic lens (e.g., microfluidic lens) system. Thelight sensor can take a variety of forms, for example, that of acomplementary metal-oxide-semiconductor (CMOS) or that of a chargecoupled device (CCD) sensor. In the present embodiment, the fluidic lenssystem of the miniature camera 2 affects different color wavelengthsnon-uniformly. Although in the present embodiment a fluidic lens systemis employed, in other embodiments other types of lens systems includingother tunable lens systems can be employed instead (or in addition to) afluidic lens system. Typically the lens system employed will afford thecamera with at least some zooming capability. Further, the control unit11 and piezoelectric (PZT) operator control unit 12 can take a varietyof forms depending upon the embodiment. In one embodiment, each of thecontrol units 11, 12 includes a processing device (e.g., amicroprocessor) and a memory device, among other components. Also, whilethe control and processing unit 10 is shown to include both the controlunit 11 and the PZT operator control unit 12, in other embodiments thefunctioning of those two control units can be performed by a singledevice.

During operation, the miniature camera 4 of the camera system 2 acquiresimages (e.g., acquired image data) of the desired region of interest 16within the field of view 14, which are transmitted to the control andprocessing unit 10. In one embodiment, the miniature camera 4 transmits,over 3 Channels, two video images and additionally the optical zoomstate. The control unit 11 (e.g., a master computer) of the control andprocessing unit 10 processes the received data, and can further transmitto the light source assemblies 6, 8 any changes that should be made intheir operation so as to adjust the intensity and/or direction/angle ofthe light emitted from the light source assemblies. Among other things,the control and processing unit 10 (particularly the control unit 11)provides color corrections to the acquired images using a deblurringalgorithm, as more fully described below.

Although shown with the single miniature camera 4 and the two lightsource assemblies 6, 8, the system can in other embodiments include anyarbitrary number of light source assemblies and any arbitrary number ofcameras. Although only a single light source assembly can be used insome embodiments, the use of the multiple light sources 6, 8 so as toprovide illumination from different angles in the present embodimentdoes afford greater efficiencies. Further, the use of multiple lightsources from different angles can make possible additional types ofimage processing. For example, different lighting angles and shadows canbe processed to create enhanced three-dimensional images and anatomicallandmarks identification.

Still referring to FIG. 1, in the present embodiment a two-way wirelesscommunication network having wireless communication links 18 is providedthat allows for the transmission of control and/or monitoring signalsbetween/among components of the system 2. The wireless communicationlinks 18 among other things allow for communication between the controlunit 11 and the miniature camera 4. More particularly in this regard,the wireless communication links 18 allow for signals to be transmittedfrom the miniature camera 4 back to the control unit 11, particularlysignals representative of video images from inside the abdominal cavityas detected by the miniature camera. Also, the wireless communicationlinks 18 allow for control signals to be provided from the control andprocessing unit 10 to the miniature camera 4 for the purpose ofgoverning the pan, tilt, and zoom of the camera lens. Optical zoominformation and any camera motion information can be further taken intoaccount by the control unit 11 in its processing of image data.

Additionally, the wireless communication links 18 allow forcommunication between the control unit 11 and the light sourceassemblies 6, 8 so as to allow for control over the on/off status,brightness, beam orientation and/or other operational characteristics ofthe light source assemblies. In at least some embodiments, the controland processing unit 10 receives video imaging and zoom data, processesit, and according to the change of optical zoom and motion estimation,transmits appropriate control signals to the lens and the light sourceassemblies.

Although not shown, it will be understood that the control andprocessing unit 10 as well as the components internal to the abdominalcavity 1 (or other body cavity or other space), such as the miniaturecamera 4 and the light source assemblies 6, 8, can each include awireless transceiver or other conventional wireless communicationshardware allowing for the establishment of the wireless communicationlinks 18. The use of the wireless communication links 18 for the purposeof allowing for communication of control and monitoring signals betweenthe control and processing unit 10 (and particularly the control unit11) and each of the miniature camera 4 and light source assemblies 6, 8eliminates the need for cables extending through the wail of the abdomen1 for such purpose.

In at least some embodiments, the control and processing unit 10coordinates all wireless communications between the components of thecamera system 2. Further, in at least some embodiments, some or allwireless communications are achieved by way of conventional signalprocessing algorithms and networking protocols. For example, in oneembodiment, the miniature camera 4 transmits to the control andprocessing unit 10 in three channels the two video images (for example,the G and B channels) and the optical zoom state. Further for example,in one embodiment, the wireless communication links 18 employ Wi-Fi carBluetooth communication protocols/technologies. Additionally, dependingon video resolution, frame rate, the distance between the control andprocessing unit 10 and the miniature camera 4 (or cameras), as well asthe number of cameras in circumstances where more than one such camerais used, it can become desirable to compress the video bit streams.Possible compression algorithms include, for example, the simple imagecoding PEG standard, the high-end image coding JPEG2000 standard, andthe high-end video coding H.264 standard, as well as Scalable VideoCoding. The algorithm choice depends on other factors such as algorithmcomplexity, overall delay, and compression artifacts.

Additionally, it will be understood that the portions of the camerasystem 2 within the abdominal cavity 1 (again, the miniature camera 4and the light source assemblies 6, 8) are autonomous and freely movablewithin the abdominal cavity, which among other things allows for thedesired region of interest 16 to be viewed from multipledirections/angles. Allowing for multiple angles of views can bedesirable in a variety of circumstances, for example, when the desiredregion of interest 16 includes multiple target organs, or target organswith complicated surface shapes. In at least one embodiment, an operator(e.g., a physician) using the PZT operator control unit 12 is able toprovide instructions to the control and processing unit 10 (the PZToperator control unit can include input devices allowing for theoperator to provide such instructions), based upon which the control andprocessing unit 10 in turn causes changes in the position(s) of one orboth of the light source assemblies 6, 8 in relation to (and, in somecases, so as to become closer to) the desired region of interest 16.Such modifications in the position(s) of the light source assemblies 6,8 can in some circumstances save energy and provide a longerillumination period notwithstanding battery powering of the light sourceassemblies.

In at least some embodiments, the particular positions of the lightsource assemblies 6, 8 and the miniature camera 4 are set up at aparticular time during an operation (e.g., by insertion into theabdominal cavity 1 by a physician early on during an operation) and thenthose positions are fixed during the remainder of the operation. In suchcases, the relative locations of the light source assemblies 6, 8 andthe miniature camera 4 can be computed at an initial set up stage. Theinitial parameters for the light beams 11, 13 provided by the lightsource assemblies 6, 8 (e.g., angles and beam direction) as well as forthe miniature camera 4 (e.g., viewing angles with respect to the lightbeams) are also computed. In some embodiments, additional movements ofthe light source assemblies 6, 8 and/or the miniature camera 4 can alsobe tracked. In some such embodiments, to calculate the light anglemodification (or a desired light angle modification), ongoinginformation regarding the spatial locations of the miniature camera 4,light source assemblies 6, 8, and in at least some cases also thecontrol and processing unit 10 is obtained by triangulation of thewireless signals captured from each component using any of variousconventional triangulation techniques. Additionally as discussed infurther detail below, in at least some embodiments, the illuminationemitted from the light source assemblies 6, 8 can further be adjusted asnecessary.

Given the particular features of the miniature camera 4 as describedabove and further below, the miniature camera provides severaladvantageous features. In particular, in the present embodiment theminiature camera 4 is small in size, has powerful optical zoomcapabilities, and has few or no moving parts while zooming. The powerfuloptical zooming afforded by the miniature camera 4 allows the miniaturecamera to be distant from the desired region of interest 16.Consequently, while if the light source assemblies 6, 8 were physicallyconnected to the miniature camera 4, a significant amount ofillumination would potentially be wasted (in that it would not reach thedesired region of interest 16), in the present this need not occur.Rather, instead of placing the light source assemblies 6, 8 on (orotherwise physically connecting those assemblies to) the miniaturecamera 4, in the present embodiment the light source assemblies areseparate from and can be moved independently of the miniature camera 2.

Turning now to FIGS. 2A-2C, one of the light source assemblies 6, 8,namely, the light source assembly 6, is shown schematically in moredetail (in cross-section) in three different operational circumstances.It will be understood that, in the present embodiment, each of the lightsource assemblies 6, 8 is identical in construction and operation,albeit in other embodiments the various light sources need not beidentical. As shown in each of FIGS. 2A-2C, the light source assembly 6includes a tunable lens that in the present embodiment is a microfluidiclens 20 (albeit in other embodiments other types of tunable lenses canbe employed as well), and additionally a light emitting diode (LED) 22.The LED 22 is small (typically about 1 mm² including package) and can bea widely available, off the shelf component that is of low cost, andthat can easily be mounted on a printed circuit board (also not shown)so as to form a surface-mount LED or the like. Although not shown, thelight source assembly 6 further should be understood as including apower source such as a battery. Having an integrated power source tosupply power to the light source assembly 6 is advantageous because, byusing such a structure, there is no need to employ cables going throughthe abdominal wall into the abdominal cavity 1 (albeit this limits theamount of power available and further increases the need for focusingthe light to the desired region of interest 16 rather than illuminatinga large unrecorded field beyond the desired region of interest).

In the present embodiment, the microfluidic lens 20 (and, indeed, anymicrofluidic lens or lenses employed in the camera 4 as well) can be amicrofluidic lens such as that described in U.S. patent application Ser.No. 11/683,141, which was filed on Mar. 7, 2007 and is entitled “FluidicAdaptive Lens Systems and Methods” (and which issued as U.S. Pat. No.7,453,646 on Nov. 18, 2008), which is hereby incorporated by referenceherein. The microfluidic lens 20 is placed a few millimeters away fromthe LED 22 (although this separation distance is not shown FIGS. 2A-2C)and has a tunable focal length and appropriate aperture typicallyseveral times the dimension of the LED die. For example, if the LED dieis 1 mm×1 mm (i.e., 1 mm²), the clear aperture of the tunable lens canbe 3 mm by 3 mm (i.e., 9 mm²) or greater. By adjusting the focal lengthof the microfluidic lens 20, the radiation pattern (e.g., radiationangle) of each light source can be dynamically adjusted to obtainoptimal illumination to match varying fields of view of the camera. Thatis, the light beam 11 emanating from the light source assembly 6 can bevaried in this manner.

To illustrate this effect, FIGS. 2A, 2B and 2C in particularrespectively show medium, narrow and wide radiation patterns(particularly the extent of an angle θ_(FWHM) corresponding to fullwidth at half maximum intensity) corresponding to three versions of thelight beam 11 that are generated by the same light source assembly 6when the microfluidic lens 20 is tuned to three different settings. Moreparticularly as shown, the narrow and medium radiation patterns areachieved when the microfluidic lens 20 is tuned so as to behighly-convex and moderately convex, respectively, while the wideradiation pattern is achieved when the microfluidic lens is tuned so asto be concave. In the present embodiment, the distance between the LED22, and the microfluidic lens 20 in the light source assembly 6 is closeenough (e.g. 3-5 mm) so that, even for the shortest focal distance ofthe lens, the light of the LED is “defocused” rather than being“focused”. The defocused LED light radiates at a divergent angle.

Notwithstanding the exemplary radiation patterns shown in FIGS. 2A-2C,assuming that the microfluidic lens 20 has an ultrawide tuning range,the radiation angle of illumination can be controlled to vary within awide range of approximately 15 degrees to 140 degrees. The optimalillumination condition is achieved when the radiation pattern producedby the light source assembly 6 is about 20% greater than thefield-of-view 14 of the miniature camera 4. For a 4× optical zoom camerawith a field-of-view of 25 to 100 degrees, the corresponding desiredillumination angles range from 30 to 120 degrees, well within thecapability of the above-described light source assembly 6. As a result,for any chosen field of view of a miniaturized camera, the subjectmatter within the field of view will be uniformly illuminated for thebest image quality.

Turning to FIGS. 3A-3C, another embodiment of a light source assembly 26is shown in simplified schematic form a in three different views (againin cross-section). The light source assembly 26 can be used in place ofeither of the light source assemblies 6, 8 discussed above. The lightsource assembly 26 in particular allows for steering of an illuminationbeam 28 emanating from the light source assembly, so as to allow forefficient illumination of various different desired regions of interest(e.g., the region of interest 16) without requiring mechanical movementof the light source so that it can shine upon those different regions.In some circumstances, such steering of the illumination beam 28 alsoallows for efficient illumination of a given desired region of interestfrom different directions/angles.

While in this embodiment the light source assembly 26 (like the lightsource assembly 6 of FIGS. 2A-2C) again includes both a tunable fluidic(again in this embodiment, microfluidic) lens 24 and a LED light source,in this embodiment the LED light source includes not merely one LED butrather includes an array of multiple LEDs 22 adjacent the lens 24.Depending upon the embodiment, the array 22 can take a variety of forms,employ any arbitrary number of LEDs, and/or employ LEDs of any arbitrarycolor(s). In the present exemplary embodiment, the array 22 particularhas a center LED 30 aligned with a central axis 32 of the lens 24, andsix additional LEDs 34 arranged along a concentric ring extending aroundthe center LED 30 in manner where each of the additional LEDs is spacedapart by the same distance from each of its three neighboring LEDs (thatis from the center LED and from each of the neighboring LEDs along theconcentric ring). Thus, in the cross-sectional views provided by FIGS.3A-3C, the center LED 30 and two of the six additional LEDs are visible(it will be understood that the remaining four additional LEDs would before or aft of the three visible LEDs, that is, into or out of the pagewhen viewing FIGS. 3A-3C).

As further shown in FIGS. 3A-3C, by selectively turning on or off thedifferent ones of the LEDs so that the different LEDs are illuminated,the same light source assembly 26 given the same tuning of the lens 24is capable of producing light beams 28 that have different orientations.More particularly, as shown in FIG. 3B, when the center LED 30 is turnedon and the additional LEDs 34 are all shut off, the light beam 28 isaligned (that is, centered about) the central axis 32 of the lens 24. Bycomparison, as shown in FIGS. 2A and 2C, given proper design of the lensaperture and the f-number of the lens 24, when the respective additionalLEDs 34 on the concentric ring are energized (and the center LED 30 isshut off), the light beams 28 produced are light cones that areangularly offset from the central axis 32 (by amounts θ_(OFF)). Moreparticularly, FIG. 2A illustrates a circumstance where the additionalLED 34 to the left of the center LED 30 is energized, while FIG. 2Cillustrates a circumstance where the additional LED to the right of thecenter LED is energized. The amount of angular offset in any givencircumstance is determined by the particular spacing of the respectiveadditional LEDs 34 (and also possibly, to some extent, by the tuning ofthe lens 24). As discussed with respect to FIGS. 2A-2C, the particularangles of divergence θ_(FWHM) of the light beams 28 are also determinedby the tuning of the lens 24. That is, the light beams 28 are of atunable divergent angle.

Given that different orientations of the light beams 28 can be achievedby energizing different ones of the LEDs 30, 34, it will be understoodthat beam steering can be achieved in the present embodiment withoutmechanical movement by selecting (and varying) which one(s) of the LEDs30, 34 on or within the concentric ring are powered at any given time.While in the above example, only a single one of the LEDs 30, 34 isenergized at a given time, in other embodiments multiple LEDs can alsobe powered simultaneously to create multiple beams along differentdirections, which can be desirable depending upon the particular desiredregions of interest that are desirably illuminated, or to provideillumination suitable for supporting multiple cameras. As already noted,any of a variety of LED arrays having a variety of formations with anyarbitrary number of LEDs can be utilized depending upon the embodiment.In other embodiments, additional LEDs and/or additional tunable lensescan be added to the assembly to achieve more finely-graded orquasi-continuous beam steering. That is, as the number of LEDs increasesand/or the spacing between LEDs decreases, there is an increased abilityto achieve finer steering of the light beams that are generated. Suchquasi-continuous designs are more efficient and can allow for energysaving. Instead of illuminating a field that is composed of the marginsbetween two LED areas by these two LEDs, the quasi-continuous designwill illuminate the same area using only one LED.

Further, depending upon the embodiment, more than one lens can beutilized. For example, two or more lenses can be positioned sequentiallybetween the LED(s) and the outer surface of the light source assemblythrough which emitted light leaves the light source assembly.Additionally, in the case of two or more lenses, the lenses can operateas a zoom lens system but in a reverse sense. Like a zoom lens capableof continuously varying its magnification factor, a dual tunable-lenssystem can continuously vary the orientation of the radiation patternwithout physically tilting the device. Through the use of multiplelenses in addition to multiple LEDs (particularly as LED array with alarge number of LEDs), truly, continuous beam steering can be achieved.Again, such truly continuous beam steering can be more energy-efficientin providing desired illumination.

To summarize, as illustrated by FIGS. 2A-3C, in at least someembodiments of the present invention the emitted light can be controlledin two respects, namely, the area and the location of illumination. Inat least some embodiments, in order to control the area of illumination,a fluidic tunable lens positioned in front of the light source iscontrolled. Depending on the optical zoom used by the camera, theemitted light will be adjusted to a narrow or wide radiation pattern bychanging the shape of the lens. Further, in at least some embodiments,in order to control the location of the center of the beam of emittedlight relative to a center axis of the lens, control signals can begenerated to select and power desired LEDs in an LED array. Byappropriately turning on and off different LEDs of the LID array, and/orappropriately tuning one or more microfluidic lenses, the beam emanatingfrom the light source assembly can be steered. Further, by tracking thelocation of the camera and the light source assemblies (using thecontrol and processing unit 10), both the radiation pattern and thedirection of the light beam can be varied in a manner suited forilluminating arbitrary desired regions of interest.

As mentioned above, the camera 4 of the camera system 2 in at least someembodiments includes a tunable fluidic lens (for example, a microfluidiclens) system, and can be either a still camera or a video camera foracquiring a sequence of color images. The camera 4 includes an imagesensor such as a CMOS sensor for acquiring images in each of a pluralityof color channels, such as a red color channel (R), a green colorchannel (G), and a blue color channel (B). The images can be in the formof image data arrays, each corresponding to a respective one of thecolor channels and each comprising a plurality of pixels, with eachpixel having an associated image value. As described below, some of thecolor channel images are blurred. Data from the image sensor can betransferred to the control and processing unit 10 to be furtherprocessed, and the corresponding images from each color channel can becombined to form a composite color image. The control and processingunit 10 can therefore perform a variety of image processing algorithms,including deblurring algorithms, which operate to enhance or correct anyblurred images.

In particular, because the fluidic lens system of the camera affectsdifferent color wavelengths of light non-uniformly, different colorwavelengths are focused at different focal depths resulting in thedifferent R, B color channels having different amounts of blurring. Thefluidic lens system can also cause non-uniform blurring in the spatialdomain, making objects at the center of the field of view more blurredthan objects near the outer borders. Thus, image enhancement andcorrection algorithms are desirable.

Thus, the control and processing unit can be programmed to performvarious image processing algorithms, including an image processingtechnique for correcting for image warping and a technique forcorrecting blurred images. A warping correction technique can model theimage warping taking into account both tangential and radial distortioncomponents, a set of calibration parameters can be determined and thedistortion can be inverted.

Various known deblurring techniques exist, such as Lucy-RichardsonDeconvolution and Wiener filtering techniques, which have been used tocorrect for blur that occurs in glass lens systems. Both of thesetechniques require that an appropriate point spread function (PSF) becalculated, which can be a function of color, frequency, depth, andspatial location. Calculation of the PSF can be complex, cumbersome, andpotentially inaccurate, making these techniques disadvantageous.Further, both techniques assume that the level of blur is the same foreach of the color channels and do not account for the variations in blurbetween the color channels.

Generally, the deblurring methods described herein rely on therealization that edges in natural images occur at the same location inthe different color channels. By adjusting the fluidic lens systemand/or image sensor of the camera such that one color channel is sharp(at least relative to the others) even though the other color channelsare blurred, it is possible to extract the edge information from thesharp image and use it in conjunction with non-edge information of theblurred image to produce a deblurred image. This is possible because theblurred color channels have good shading information but poor edgeinformation.

In one embodiment, the fluidic lens system and/or CMOS sensor areadjusted such that the green channel is sharp and the red and bluechannels are blurred, i.e., corresponding images are out of focus andhave more blurring distortion. Then the acquired images from the cameracan be processed (such as by the control and processing unit) to extractthe edge information from the green image, and use it to correct theblurred images corresponding to each of the red and blue color channels.The images corresponding to each color channel can then be combined toproduce a deblurred composite color image.

Basically, the image corresponding to the green color channel and theimage corresponding to the red (or blue) color channel are bothdecomposed by filtering and downsampling using a filter bank that allowsfor the separation of the edge and the shading information. Edgeinformation from the sharp green image is then meshed with the non-edgeinformation for the red (or blue) image to form a set of sub-band outputcoefficients, and these are input to a reconstruction portion of thefilter bank which includes upsampling, filtering and combining steps togenerate a less blurred red (or blue) image and ultimately a lessblurred composite color image. In one embodiment, decomposition isperformed using wavelet decomposition and wavelet transforms, while inanother embodiment contourlet decomposition and contourlet waveforms areused. In the case of the wavelet decomposition, the selection of whichsub-band output coefficients to use in an image reconstruction can bedetermined a priori. In this regard, the sharp image is decomposed toobtain certain sub-band output coefficients which are each assumed torepresent edge information (i.e., corresponding to sub-bands having ahigh frequency component) and the blurred image is decomposed to obtainother sub-band output coefficients corresponding to sub-bands which areassumed to not represent edge information (i.e., those sub-bands nothaving a high frequency component). In other words, the sub-bands forthese sets do not overlap. The output coefficients from each are thencombined to form an appropriate reconstruction set without anyevaluation of whether the green output coefficients actually representedge information.

In the case of contourlet decomposition, both the blurred and the sharpimages can be decomposed to generate a corresponding sub-band outputcoefficient for each sub-band in a respective set of sub-bands, and thetwo sets of sub-bands can be overlapping sets. Then the resultantsub-band output coefficients corresponding to the sharp image can beevaluated to distinguish between strong and weak edges. For example, anant colony optimization technique can be utilized to determine edgeinformation, although other edge detection techniques can also beemployed, in this case, a set of sub-band output coefficientscorresponding to the blurred image are modified to replace some of thesub-band output coefficients in the set with corresponding sharp imagesub-band output coefficients which correspond to edge information forthe sharp image. Some of the sub-band output coefficients in the set arenot replaced but are retained, and these are sub-band outputcoefficients which correspond to non-edge information. The resultantmodified set of sub-band output coefficients can then be used in thereconstruction process.

With respect to the wavelet decomposition and reconstruction, in oneembodiment, the following steps are preformed:

-   -   1. Select a modified perfect reconstruction filter bank;    -   2. Decompose the images into sub-bands by first filtering and        down-sampling the rows of the image, then filtering and        down-sampling the columns;    -   3. Form a set of reconstruction coefficients for the blue image        including the B^LL coefficient and the band pass sub-band output        coefficients corresponding to the green image (denoted by G^LH,        G^HL, and G^HH) instead of using the band pass sub-band output        coefficients of the blue image (denoted by B^LH, B^HL, and        B^HH);    -   4. Depending on the degree of blur, introduce more levels of        decomposition by further down-sampling and filtering the B^LL        component. The green color sub-band output coefficients can        replace more of the corresponding blue color sub-band output        coefficients; and    -   5. Reconstruct by up-sampling and filtering.

More specifically, the control and processing unit 10 performs such awavelet sub-band meshing image processing method using a modifiedperfect reconstruction filter bank to separate image edge informationand shading information. With reference to FIG. 4, an understanding of amodified perfect reconstruction filter bank begins with an understandingof a perfect reconstruction filter bank 40. In this case, this filterbank 40 has as its input a signal denoted by B^, which represents ablurred blue image from the blue color channel after passing through alens, where L0 represents a lens blurring function and B represents anunblurred blue image. Perfect reconstruction filter bank 40 includes adeconstruction portion 42 on the left side, a reconstruction portion 44on the right side, and an intermediate section 43 at which sub-bandoutput coefficients are output from the deconstruction portion 42 andinput to the reconstruction portion 44.

The filter bank 40 operates to deconstruct signal B^ into apredetermined number of sub-band output coefficients corresponding tosub-bands in a wavelet domain (akin to a frequency domain) using thedeconstruction portion 42. The filter bank then operates to reconstructa version of the signal B^ using these coefficients as input to thereconstruction portion 44, with a so-called “perfect” reconstructionupon appropriate selection of filter characteristics.

As illustrated, the decomposition portion 42 of the perfectreconstruction filter bank 40 includes two cascaded levels anddecomposes the input signal into four sub-bands. Specifically, at eachof the two different levels, decomposition of the signal B^ occurs byfiltering (using deconstruction filter H0, a low pass filter, ordecomposition filter H1, a high pass filter) and downsampling (by afactor of two) to generate a respective sub-band output coefficient foreach of the four different sub-bands at the intermediate section 43. Forexample, B^LH represents a sub-band output coefficient after filteringand down-sampling B^ twice, where L represents a low pass filter and Hrepresents a high pass filter. The resultant sub-band outputcoefficients for the illustrated filter bank include B^LL, whichrepresents the shading information, and B^LH, B^HL, and B^HH, whichrepresent the edge information.

These sub-band output coefficients are input to the reconstructionportion 44 and up sampled (by a factor of two) and filtered (usingreconstruction filters F0, a low pass filter, or reconstruction filterF1, a high pass filter) and combined, in each of two levels, toreconstruct the image signal B^. The filters H0 (low pass), F0, and H1(high pass), F1 make up a set of perfect reconstruction filter bankpairs and appropriate selection of these filters can occur using knownmethods along with the constraints described below.

As shown in FIG. 5, a modified perfect reconstruction filter bank 50also takes in image G, corresponding to a sharp image of the green colorchannel. This filter bank acts to at least partially decompose bothblurred image B^, corresponding to the blurred image data of the bluecolor channel, and also image G, corresponding to the sharp image fromthe green color channel. Sharp image G has also passed through the lensresulting in what can be denoted G^, but because little blurring occurs,it can be assumed that G^ is approximately the same as 0. Blurred imageB^ is decomposed over two levels using two low pass filters to extractits corresponding shading information in the form of a sub-band outputcoefficient denoted by B^LL. Further the filter bank 50 acts to at leastpartially decompose sharp image data G (corresponding to the sharp imagedata of the green color channel) to extract its corresponding edgeinformation in the form of the sub-band output coefficients denoted byG^HL, G^LH, and G^HH. The sub-band output coefficients B^LL, G^HL, G^LH,and G^HH form a reconstruction coefficient set which is input to areconstruction portion 54 which is the same as the reconstructionportion 44 of the perfect reconstruction filter bank 40. A new deblurredimage denoted by A* is then reconstructed by using this reconstructionset by upsampling, filtering, and combining over two levels. Image A* isan improvement over blurred image B^ and maintains the shadinginformation of the blurred blue image but has sharper edges. By usingthis modified filter bank 50, the color image edges can be improvedwithout introducing false colors.

Refinements can be made by adding more cascaded levels to thereconstruction filter bank and one or more pre-filters based on thechannel characteristics. As described below, a prefilter such as W0 canbe added to improve results, such as to filter the image prior todecomposition. Further, the number of cascaded levels in the filter hankcan be determined based on the frequency response of the initial system(filter bank and lens).

An analysis of a one-dimensional version of the system can be describedwith reference to FIGS. 6 and 7. Although this analysis applies to the1-D case, these concepts can be easily extended to the 2-D case. Thisanalysis assumes that all the filters have unit gain. FIG. 6 illustratesa 1-D standard perfect reconstruction filter bank 60, and FIG. 7illustrates a modified prefect reconstruction filter bank 70. In bothFIGS. 6 and 7, L0(z) models the blurring effects of the lens on the trueblue image B(z) as a low pass filter according to: B^(z)=L0(z)B(z).

After the lens, FIG. 6 shows that a standard filter bank can beexpressed as follows:B^rL(z)=½F0(z)[H0(z)B^(z)+H0(−z)B^(−z)]B^rL(z)=½F1(z)[H1(z)B^(z)+H1(−z)B^(−z)]

-   -   where:    -   B^rL is a reconstructed output in a low frequency sub-band (L)        of the blurred image B^.

The filter bank of FIG. 6 is modified by replacing B^rH(z) with G^rH(z)from the green image sub-band, such as shown in FIG. 7, and expressedby:G^rH(z)=½F1(z)[H1(z)G^(z)+H1(−z)G^(−z)]

-   -   where:    -   G^rH is a reconstructed output in a high frequency sub-band (H)        of the image G^.

In order to reconstruct the original image data B, an estimate for thehigher sub-band used in reconstruction must be close to the highersub-band of the original B image data. From the optical properties ofthe lens, assume that G^ better estimates the edges of the original bluesignal, as follows:H1(z)G^(z)+H1(−z)G^(−z)+E _(G) =H1(z)B(z)+H1(−z)B(−z)H1(z)B^(z)+H1(−z)B^(−z)+E _(B) =H1(z)B(z)+H1(−z)B(−z)

-   -   where E_(G) represents the error in the estimate of image G, and    -   E_(B) represents the error in the estimate of image B.

The above two equations represent estimates of the true high passsub-bands of B, where E_(G) and E_(B) are the errors of the twoestimates. Because of high edge correlation, this model assumes that theabsolute value of E_(G) is less than or equal to absolute value ofE_(B).

The green color sub-band outputs are used to create a reconstructedimage A*(z), where:

$\begin{matrix}{{A^{*}(z)} = {{B\hat{}{{rL}(z)}} + {G\hat{}{{rH}(z)}}}} \\{= {{{1/2}\{ {F\; 0{(z)\lbrack {{H\; 0(z){B\hat{}(z)}} + {H\; 0( {- z} ){B\hat{}( {- z} )}}} \rbrack}} \}} +}} \\{{{1/2}\{ {F\; 1{(z)\lbrack {{H\; 1(z){B\hat{}(z)}} + {H\; 1( {- z} ){B\hat{}( {- z} )}}} \rbrack}} \}} - {{1/2}F\; 1(z)E_{G}}} \\{= {1/{{2\begin{bmatrix}{{F\; 0(z)H\; 0(z)L\; 0(z){B(z)}} + {F\; 0(z)H\; 0( {- z} )L\; 0( {- z} ){B( {- z} )}} +} \\{{F\; 1(z)H\; 1(z){B(z)}} + {F\; 1(z)H\; 1( {- z} ){B( {- z} )}} - {F\; 1(z)E_{G}}}\end{bmatrix}}.}}}\end{matrix}$

It is desirable to remove the aliasing component B(−z) and reconstruct adelayed version of the original signal B(z). This leads to the followingreconstruction conditions:F0(z)H0(z)L0(z)+F1(z)H1(z)=2z e ⁽⁻¹⁾;F0(z)H0(−z)L0(−z)+F1(z)H1(−z)=0; and|(F1(z)E _(G))|=ε

H0(z) and L(z) are both low pass filters. If H0(z) has a lowertransition frequency than that of L0(z), then the following twoapproximations hold:H0(z)L0(z) is approximately equal to H0(z), andH0(−z)L0(−z) is approximately equal to H0(−z).

These approximations simplify the three reconstruction conditions aboveto:F0(z)H0(z)+F1(z)H1(z) is approximately equal to 2z e ⁽⁻¹⁾;F0(z)H0(−z)+F1(z)H1(−z) is approximately equal to 0; and|(F1(z)E _(G))|=ε

Where the equality holds for the above three conditions, theseconditions match the perfect reconstruction conditions of a conventionaltwo-channel filter band. If the system uses a perfect reconstructionfilter bank, then the output is simplified as follows:A*(z)z ⁽⁻¹⁾ B(z)−½F1(z)E _(G)

This process will lead to an A*(z) that closely resembles the signalB(z) with a small error factor E_(G). L_(G) changes spatially, and thebetter results will be obtained in regions where the two images havehigh edge correlation.

This method assumes that the lens can be modeled as a low-pass filter onthe blue color channel. An analysis of the error begins with therealization that the process requires H0(z) to have a lower transitionfrequency than L0(z) so that the approximations set forth above aresatisfied. For a two channel perfect reconstruction filter bank, H0 hasa transition band centered at ω=0.5 π. If it is determined that theL0(z) transition frequency of the lens is less than 0.5 π, thenadditional levels of sub-band decomposition can be added to the filterbank 50.

The above analysis suggests that there is a trade-off between the errorcreated by using the green sub-band outputs and the error created by thelens when blurring the blue sub-bands. Rather than recovery of the highfrequencies, the deblurring method replaces this part of the spectrumwith the corresponding green sub-hand output coefficients, although whenthe low-pass lens filter no longer passes the higher frequencysub-bands, then the trade-off favors using the green sub-bands. Thelevels of decomposition, c, can be increased until the transitionfrequency of the blurring filter falls beyond the transition frequencyof the lowest sub-band.

Determining the levels of decompositions involves the followingconsiderations. Recall that L0(e^(jω)) represents the blur filter of theblue image. An appropriate frequency response is as follows: L0(e^(jω))is approximately 1 if |ω|<π4, and is approximately 0 if π/4≦|ω|<π.

Thus, the method should use the blue sub-bands for all frequencies|ω|<π/4 that the lens passes, and use the green sub-bands for allfrequencies π/4≦|ω|<π. For this L0(e^(jω)), the method requires a twolevel decomposition, such as shown in FIGS. 5 and 7. Further:

${B\hat{}{{LL}(z)}} = {{1/{2\lbrack {H\; 0( z^{1/2} ){1/2}( {{H\; 0( z^{1/4} ){B\hat{}( z^{1/4} )}} + ( {H\; 0( {- z^{1/4}} ){B\hat{}( {- z^{1/4}} )}} )} )} \rbrack}} + {1/{2\lbrack {H\; 0( {- z^{1/2}} ){1/2}( {{H\; 0( {j\; z^{1/4}} ){B\hat{}( {j\; z^{1/4}} )}} + ( {H\; 0( {{- j}\; z^{1/4}} ){B\hat{}( {{- j}\; z^{1/4}} )}} )} )} \rbrack}}}$

Instead of using the B^LH, B^HL, and B^HH terms, this method usesinstead the respective G^LH, G^HL, and G^HH terms. The B^LH, B^L, andB^HH sub-band terms estimate the original B image poorly. The unfilteredsub-bands terms of G^LH, G^HL, and G^HH estimate the B signal wellbecause of strong edge correlation. The reconstruction error depends onthe accuracy of the estimate of these sub-band terms.

Now consider the following change of variables:R(z)=H0(z)[B(z)−B^(z)], andP(z)=H0(z)B(z)(1−L0(z)).

E_(BLL) (z) represents the error between BLL and B^LL, and can beexpressed as follows:

$\begin{matrix}{{E_{BLL}(z)} = {{{BLL}(z)} - {B\hat{}{{LL}(z)}}}} \\{= {{1/{4\lbrack {H\; 0( z^{1/2} ){R( z^{1/4} )}} \rbrack}} + {1/{4\lbrack {H\; 0( z^{1/2} ){R( {- z^{1/4}} )}} \rbrack}} +}} \\{{1/{4\lbrack {H\; 0( {- z^{1/2}} ){R( {j\; z^{1/4}} )}} \rbrack}} + {1/{4\lbrack {H\; 0( {- z^{1/2}} ){R( {{- j}\; z^{1/4}} )}} \rbrack}}} \\{= {{1/{4\lbrack {H\; 0( z^{1/2} ){P( z^{1/4} )}} \rbrack}} + {1/{4\lbrack {H\; 0( z^{1/2} ){P( {- z^{1/4}} )}} \rbrack}} +}} \\{{1/{4\lbrack {H\; 0( {- z^{1/2}} ){P( {j\; z^{1/4}} )}} \rbrack}} + {1/{4\lbrack {H\; 0( {- z^{1/2}} ){P( {{- j}\; z^{1/4}} )}} \rbrack}}}\end{matrix}$

Thus, four distinct terms comprise E_(BLL)(z). For the first term,L0(e^(jω/4)) is approximately 1 when |ω| is less than π, thus the firstterm is approximately 0. For the second term, H0 is a low-pass filter.Thus H0(−e^(jω/4)) is approximately equal to zero by construction andthe second term is approximately 0. For the remaining two terms,H0(−e^(jω/2)) is approximately 0 and those terms are approximately 0. Inorder to make E_(BLL)(z) small, H0 should approximate an ideal low-passfilter as much as possible. This effect suggests that adding morecoefficients to the filters will improve performance. BecauseL0(e^(jω/4)) passes the frequencies in this sub-band, the estimateproduces a small error E_(BLL)(z).

Consider generalizing L0, such that L0(e^(jω)) is approximately 1 if |ω|less than ω₀, and is approximately 0 if |ω| is greater than or equal toω₀ and less than π.

In order to reduce the overall error, the decomposition level c canincrease until π/2^(c) is less than or equal to ω₀, and ω₀ is less thanor equal to π/2^(c−1). Choosing a large c means discarding parts of thefrequency spectrum which the lens does not actually corrupt. Making ctoo small will increase the error because in the low band of thefrequency spectra 0 is not equal to (1-L(z)).

In other words, the optical properties of the lens blur out B^LH, B^HL,and B^HH and yield a high approximation error. G^LH, G^HL and G^HHbetter estimate the edges of the original blue signal.E _(BLH) =BLH−B^LHE _(BHL) =BHL−B^HLE _(BHH) =BHH−B^HHE _(GLH) =GLH−G^LHE _(GHL) =GHL−G^HLE _(GHH) =GHH−G^HHHere |E _(GHH) |≦|E _(BHH) |;|E _(GHL) |≦|E _(BHL)|; and |E _(GHL) |≦|E_(BLH)|.

Recall that G^LH, G^HL and G^HH then pass through the reconstructionportion of the filter bank. Thus, the filter band must satisfy knownreconstruction conditions such as set forth in G. Strang and T. Q.Nguyen, “Wavelets and filter banks”, Cambridge, Mass.:Wellesley-Cambridge, 1997, which is hereby incorporated by referenceherein. The following expresses the output of the filter bank:A*(z)=z ⁻¹ B(z)−F0(z)F0(z ²)E _(BLL) −F0(z)F1(z ²)E _(GLH) −F1(z)F0(z²)E _(GHL) −F1(z)F1(z ²)E _(GHH)

Increasing the decomposition level c causes more E_(G) terms to appear.To reduce E_(BLL) without introducing extra E_(G) terms, level c shouldbe increased until π/2^(c)≦ω₀≦π2^(c−1).

Best results are obtained by limiting the number of E_(G) terms, andusing only the sub-bands with small E_(B) terms. Similar to above,consider the error E_(BLH) between BLH and B^LH:

$\begin{matrix}{{E_{BLH}(z)} = {{{BLH}(z)} - {{B\hat{}L}\;{H(z)}}}} \\{= {{1/{4\lbrack {H\; 1( z^{1/2} ){P( z^{1/4} )}} \rbrack}} + {1/{4\lbrack {H\; 1( z^{1/2} ){P( {- z^{1/4}} )}} \rbrack}} +}} \\{{1/{4\lbrack {H\; 1( {- z^{1/2}} ){P( {j\; z^{1/4}} )}} \rbrack}} + {1/{4\lbrack {H\; 1( {- z^{1/2}} ){P( {{- j}\; z^{1/4}} )}} \rbrack}}}\end{matrix}$

Again, this expression includes four terms. The first three terms areapproximately zero by construction because H1(e^(jω/2)) is approximatelyzero and H0(e^(jω/4)) is approximately zero. However, the last termcontains error. To reduce this error, G^LH can replace B^LH as discussedabove. For the lower frequency sub-bands, the correlation does poorlyand the reconstruction suffers.

To reduce the error further, a pre-filter W0(z) can be added directlyafter L0(z) in FIG. 5. By adding this filter, the fourth term of theprior equation changes to:¼Q(−jz ^(1/4))[1−L0(−jz ^(1/4))W0(jz ^(1/4))]where Q(z)=H1(z ²)H0(z ²)H0(z)B(z).

To make the last term approximately zero, the filter W0(z) needs tosatisfy the following:1−L0(−jz ^(1/4))W0(−jz ^(1/4))=0

Assume more is known about L0(e^(jω)), as for example that L0(e^(jω)) isapproximately:1 if |ω|<ω₀;δ if ω₀<|ω|<ω_(δ); and0 if ω_(δ)<|ω|<π.Here 0<δ<1 represents the transition band. The first zero of L0(e^(jω))has a higher frequency than ω_(δ). A modified Wiener filter can bedesigned to reduce the error in all sub-bands with frequencies belowω_(δ). The filter bank design increases the level of decomposition sothat the highest B^ sub-band used in reconstruction has a transitionfrequency that is arbitrarily close to ω_(δ). In practice, complexityand image size limit the number of levels of decomposition.

In another embodiment, a contourlet sub-band meshing method is used fordeblurring an image in a first color channel, such as a blue (or red)channel, using information from a second sharp color channel, such as agreen channel. This method is similar to the wavelet-based meshingmethod described above in that decomposition and reconstruction areinvolved. However, the contourlet sub-band meshing method uses acontourlet transform instead of a wavelet transform and generates anedge map for further analysis of the edges prior to substitution of somegreen coefficients for blue ones in the reconstruction of the deblurredblue image.

The contourlet transform was recently proposed by Do and Vetterli as adirectional multi-resolution image representation that can efficientlycapture and represent smooth object boundaries in natural images (asdiscussed in Minh N. Do and Martin Vetterli, “The contourlet transform:An efficient directional multi resolution image representation,” IEEETrans. on Image Processing, vol. 18, pp. 729-739, April 2009, which ishereby incorporated by reference herein). More specifically, thecontourlet transform is constructed as an iterated double filter bankincluding a Laplacian pyramid stage and a directional filter bank stage.Conceptually, the operation can be illustrated with reference to FIGS. 8a-8 b. A Laplacian pyramid iteratively decomposes a 2-D image into lowpass and high pass sub-bands, and directional filter banks are appliedto the high pass sub-bands to further decompose the frequency spectrum.The process is iteratively repeated using a downsampled version of thelow pass output as input to the next stages. Using ideal filters, thecontourlet transform will decompose the 2-D frequency spectrum intotrapezoid-shaped sub-band regions as shown in FIG. 8 b.

A two dimensional decomposition portion 90 of a contourlet filter bankis schematically shown in FIG. 9, and is operationally somewhat similarto the operation of the decomposition portion of the filter banksdescribed above. Using this two level contourlet decomposition filterbank for the blue channel, a decomposition of the blue image producesthree outputs: the low pass output B^LL, the band pass output B^LH, andthe high pass output B^HH. The green image is similarly decomposed togenerate G^LH and G^HH, which can be substituted for the B^LH and B^HHcoefficients as described below. Of course more levels or stages ofiteration can be added to decompose the image into many more sub-bands.The blue image and the green image are both decomposed using a desirednumber of levels or stages.

An ant colony optimization edge detection scheme of this method producesa binary edge map which is then dilated and used to decide whichsub-band output coefficients corresponding to the blue channel will bereplaced with green sub-band output coefficients and which sub-bandsoutput coefficients will not be replaced. Improved results compared toconventional methods can be obtained because the variable nature ofcontourlets allows for the natural contours of an image to be moreaccurately defined. Further, the edge detection scheme allows for thecharacterization of edges as strong or weak.

Consequently, the contourlet sub-meshing method can be performed asfollows:

1. Select a contourlet filter bank and decompose both the green and theblue image into sub-hands by filtering and down-sampling, to generatecorresponding sub-band output coefficients for each image.

2. Detect major edges in the green color channel based on an ant colonyoptimization (ACO) edge detection scheme to create a binary map in thecontourlet domain. The binary edge map can be dilated to collect thearea around detected major edges such that areas around the edges in thecontourlet domain will also take a value of 1 rather than 0.

3. Select each coefficient of the band-pass contourlet sub-bandsaccording to the dilated binary edge map, which is used to determine thebest coefficient (green or blue).

4. Depending on the degree of blur, additional levels of decompositioncan be introduced by further down-sampling and filtering the lowestfrequency sub-band. The green color sub-bands can then replace more ofthe corresponding blue color sub-bands. The lowest sub-band shouldremain the original blurred blue color sub-band in order to reduce falsecoloring.

5. Reconstruct the blue image by up-sampling, filtering, and combiningas appropriate.

Further with respect to the ACO edge detection scheme, this scheme isdescribed in an article by Jing Tian; Weiyu Yu, and Shengli Xie, titled“An ant colony optimization algorithm for image edge detection,” in IEEECongress on Evolutionary Computation, 2008, pages 751-756, which ishereby incorporated by reference herein. The ACO scheme arises from thenatural behavior of ants in making and following trails. Here, a binaryedge map for the green channel can be generated with each sub-bandhaving a value of 1 or 0 depending on whether a strong edge is presentor not. Dilation allows more of the green coefficients to be used. Letg_(em) be the dilated green edge map in the contourlet domain. Then, anew coefficient for a sub-band is generated based on this map, forexample:

$\begin{matrix}{{{ALH}(w)} = {{{B\hat{}{{LH}(\omega)}}\mspace{14mu}{if}\mspace{14mu}{g_{em}(\omega)}} = 0}} \\{= {{{G\hat{}{{LH}(\omega)}}\mspace{14mu}{if}\mspace{14mu}{g_{em}(\omega)}} = 1}}\end{matrix}$

Thus this method replaces some of the blurred blue edges with sharpgreen edges but keeps those blurred blue edges which correspond to weakgreen edges. This method assumes that a strong green edge indicates asimilarly strong true blue edge and chooses a corresponding greencoefficient. In areas with a, weak green edge, the method assumes thatthe blurred blue edge better matches the true sharp blue image andchooses the corresponding blue coefficient. Natural images usuallyadhere to this generalization and thus improved image reconstruction canbe achieved.

An appropriate dilation radius (such as in the range of 5-25 pixels) isselected. At a higher dilation radius, the border around the edgesincreases and fewer ghosting artifacts occur. Also, at a higher dilationradius, the edges will appear sharper, but the shading will differ fromthe clean image.

More particularly, mixing coefficients between the two color channelscreates ghosting artifacts. The mean squared error (MSE) between thereconstructed image and the original image becomes a function of theseartifacts:MSE_(BB^) =F(A _(ghost)(g _(em)),G _(edge)(g _(em)))

A trade off exists because the green coefficients produce sharp edges,but can result in false coloring. Mixing with blue coefficients producesless false coloring, but introduces ghosting artifacts. The goal is tominimize this trade-off and produce a natural looking image.

Consider B^LH expressed in terms of B(w1, w2). As illustrated in FIG. 10a, the band-pass output can be expressed as shown, where HL is a lowpass filter, HH is a high-pass filter, and HD is a group of directionalfilters. Only the term when m=0, n=0 matters, as all the other terms areclose to 0. The clean blue image and green image have similar BLH andG^LH terms. Thus, the method uses GLH when it has a smaller MSE asexpressed in FIG. 10 b. Many of the similar terms can combined intovariable a, as shown in FIG. 10 c, which can then be simplified as shownin FIG. 10 d. The difference of squares then results in the inequalityshown in FIG. 10 e, which leads to the two conditions shown in FIG. 10f. If either of these conditions is satisfied, then a better MSE isachieved. By construction, L0(ω1/2, ω2/2) B(ω1/2, ω2/2)=B^(ω1/2, ω2/2)is the blurred blue image obtained from the camera. The last twoconditions of FIG. 10 f can be rewritten as shown in FIG. 10 g.

Similar conditions can be produced for each of the outputs of thedeconstruction filter bank. In the high frequency sub-bands, L0(ω1/2,ω2/2) is approximately zero and blurs out these coefficients. Underthese circumstances, the expression shown in FIG. 10 f becomes:

$0 < \frac{G\hat{}( {{\omega_{1}/2} - {\omega_{2}/2}} )}{B( {{\omega_{1}/2} - {\omega_{2}/2}} )} < 2$

For the high frequency components, the numerator and the denominator ofthe expression above will typically have similar magnitude, satisfyingthe upper bound. The expression above also suggests that in the highfrequency sub-bands, the method will reduce the MSE when the green andclean blue coefficients have the same sign and a strong correlationexists between the color channels. In areas where they have differentsigns, the condition fails, and the poor correlation with result incolor bleeding.

For the lowest frequency components, the blur kernel (L0) does notaffect these frequencies, and thus L0(ω1/2, ω2/2) is approximately 1. Inthis case, B(ω1/2, ω2/2) is approximately equal to B^(ω1/2, ω2/2), andthe equations of FIG. 10 g are not satisfied. Intuitively, the blurkernel does not affect the lower frequencies and substituting in greensub-bands at these lower frequencies does not reduce the MSE.

The contourlet method is an improvement over the wavelet method.Further, as the size of the blur kernel increases, the input imagequality decreases and the PSNR (Peak Signal to Noise Ratio) decreases.With the blue image more degraded, more of the green color channelcoefficients can be used. This means that more levels of decompositionare required.

The camera system 2 has been described with respect to its use inimaging body cavities as can be employed in MIS. However, otherapplications for the camera system and color correction algorithm arealso contemplated. For example, cameras with extensive zoom control areavailable today for use in outer space for homeland security purposes.Light sources for enable the use of such cameras remains an issue inconventional designs. Either the distance or the desire to remainunrecognized prevents the use of a light source directly from thelocation of the camera. A camera system such as described hereinincluding light source assemblies and network communications can resolvethis difficulty. A powerful camera can be positioned miles away from thetarget sites as long as the light source is able to adjust itselfaccording to the movement of the camera. Wireless communication betweenthe light source assemblies and the camera can enable the use of thecamera even when very distant from the target, for example, whenairborne and mobile. Embodiments of the present invention can alsoenable various systems that use only one powerful camera that iscentrally located with the ability to acquire images from differentareas, as long as it includes a light source for each area of interest.

Also for example, embodiments of the present invention are also suitablefor various civilian purposes such as using a powerful light source withvariable illumination pattern and steering capability among differentgroups. Having a powerful light source that can be used by differentconsumers will allow landing airplanes and docking ships to use thelight source of the airport or the seaport for assistance in landing.This will enable the consumers to direct the light to where it is neededrather than have a standard stationary illumination. Further forexample, a system such as the camera system 2 can be used for imagingother spaces and regions other than cavities of the human body (orcavities in an animal body), including spaces or regions that areotherwise hard to reach.

Additionally, while much of the above description relates to the camerasystem 2, which has both the camera 4 as well as one or more of thelight source assemblies 6, 8 (as well as the control and processing unit10), the present invention is also intended to encompass otherapplications or embodiments in which only one or more of thesecomponents is present. For example, the present invention is alsointended to relate to a light source assembly that is independent of anycamera and merely used to provide lighting for a desired target. Thatis, the present invention is also intended to encompass, for example, astand-alone light source assembly such as that described above thatemploys (i) one or more tunable lenses for the purpose of controlling anoutput light pattern (e.g., amount of light divergence), and/or (ii)controllable LEDs that can be switched on and off to cause variations inlight beam direction.

Any system with high edge correlation can benefit from theabove-described deblurring algorithm. Other future applications includeuses in super resolution and video compression. For super resolution,all three color planes share the same edge information, the resultingcolor image has much sharper edges than traditional techniques. Forvideo compression, the edge information is redundant between colorplanes, such that one can use the edge information from one color planeas edge information for all three color planes. This redundancy can beused to save on the number of bits required without sacrificing much interms of quality. Another example is in systems where one sharp imagesensor can improve the quality of an inexpensive sensor such as infraredwhich has high edge information. Using the same concept of wirelessnetwork and motion estimation but instead of having a light sourcehaving a powerful weapon will enable land forces to be mobile and use aheavy weapon at the same time. This weapon could be located far from thefire zone or even airborne and could be used by multiple consumers. Theweapon can be locked into target by a camera carried by the consumer aslong as its spatial orientation is registered and communication to theweapon is available. Motion estimation and distance to the target can becalculated in the processor near the weapon.

It is specifically intended that the present invention not be limited tothe embodiments and illustrations contained herein, but include modifiedforms of those embodiments including portions of the embodiments andcombinations of elements of different embodiments as come within thescope of the following claims.

We claim:
 1. A method for deblurring a blurred first image correspondingto a first color channel of a camera that also produces a sharp secondimage corresponding to a second color channel of the camera, wherein thefirst and the second images each include a plurality of pixels with eachpixel having an associated respective value, the method comprising:decomposing the first image by filtering and downsampling to generate afirst set of one or more first sub-band output coefficients, whereineach first sub-band output coefficient corresponds to a respectivesub-band in a selected one of a wavelet and a contourlet domain;decomposing the second image by filtering and downsampling to generate asecond set of second sub-band output coefficients, wherein each secondsub-band output coefficient corresponds to a respective sub-band in theselected domain; selecting those second sub-band output coefficientsthat represent edge information, each of the selected second sub-bandoutput coefficients corresponding to a respective selected sub-band,with the selected sub-bands together defining an edge sub-band set;preparing a third set of sub-band output coefficients which includes theselected second sub-band output coefficients representing edgeinformation and at least one of the first sub-band output coefficientscorresponding to a sub-band other than those sub-bands in the edgesub-band set; and reconstructing a deblurred first image by upsamplingand filtering using the third set of sub-band output coefficients asinput.
 2. The method of claim 1, wherein each decomposing step occursover two or more levels and the reconstructing step occurs over the samenumber of levels.
 3. The method of claim 2, wherein the number of levelsof decomposition is determined based at least in part on a frequencyresponse of a blurring lens of the camera.
 4. The method of claim 1, thefirst set and the second set are combined to form the third set.
 5. Themethod of claim 1, wherein the sub-bands corresponding to first set andthe second set are predetermined and non-overlapping.
 6. The method ofclaim 1, further including prefiltering the second image prior todecomposition of the second image.
 7. The method of claim 6, wherein theprefiltering uses a Wiener filter.
 8. The method of claim 1, wherein thedecomposition steps occur in the contourlet domain and the selectedsecond sub-band output coefficients are those which have been determinedto represent edge information.
 9. The method of claim 8, wherein theedge information is determined using an ant colonization optimizationscheme.
 10. A method for deblurring a first image of a camera, whereinthe camera generates the first image corresponding to a first colorchannel and a second image corresponding to a second color channel,wherein the first and the second images each includes a plurality ofpixels with each pixel having an associated respective value, the methodcomprising: selecting a filter bank having a decomposition portion and areconstruction portion, the decomposition portion having at least twocascaded levels for receiving two inputs and generating intermediatesub-band output coefficients for each of a predetermined number N ofsub-bands, the reconstruction portion having at least two cascadedlevels for receiving the intermediate sub-band output coefficients andgenerating an output, wherein the decomposition portion includesmultiple decomposition stages at a first level connecting to Ndecomposition stages at a second level, each decomposition stageincluding at least one of a decomposition filter and a downsampler,wherein the reconstruction portion includes N reconstruction stages at athird level connecting to N/2 reconstruction stages at a fourth level,each reconstruction stage including at least one of an upsampler and areconstruction filter; decomposing the first image using a first part ofthe decomposition portion of the filter bank to generate at least onefirst intermediate sub-band output coefficient corresponding to one ofthe N sub-bands; decomposing the second image using a second part of thedecomposition portion of the filter bank to generate at least a second,a third intermediate, and a fourth sub-band output coefficient, eachcorresponding to a respective one of the N sub-bands; and reconstructinga deblurred image corresponding to the first color channel using atleast the first, the second, the third, and the fourth sub-band outputcoefficients as input to the reconstruction side of the filter bank,wherein each decomposing step occurs over two or more levels and thereconstructing step occurs over the same number of levels and the numberof levels of decomposition is determined based at least in part on afrequency response of a blurring lens of the camera.
 11. The method ofclaim 10, wherein the sub-bands corresponding to intermediate sub-bandoutput coefficients of the first image are non-overlapping with thesub-bands corresponding to the intermediate sub-band output coefficientsof the second image.
 12. The method of claim 10, further includingprefiltering the second image prior to decomposition of the second colorimage.
 13. The method of claim 12, wherein the prefiltering uses aWiener filter.
 14. A method for deblurring a blurred first image,wherein the blurred first image corresponds to a first color channel ofa camera and a second image corresponds to a second color channel of thecamera, wherein the first and the second images each comprise aplurality of pixels with each pixel having an associated respectivevalue, the method comprising: decomposing the first image by filteringand downsampling in a decomposition portion of a filter bank to generatefor a predetermined number of sub-bands in a contourlet domain a firstset of first sub-band output coefficients, wherein each first sub-bandoutput coefficient corresponds to a respective one of the sub-bands;decomposing the second image by filtering and downsampling in thedecomposition portion of the filter bank to generate for thepredetermined number of sub-bands a second set of second sub-band outputcoefficients, wherein each second sub-band output coefficientcorresponds to a respective one of the sub-bands in the contourletdomain; determining which of the second sub-band output coefficients andcorresponding respective sub-bands represent edge information; preparinga set of third sub-band output coefficients by modifying the first setof first sub-band output coefficients to replace each of those firstsub-band output coefficients in the first set which correspond to therespective determined sub-bands representing edge information with thecorresponding second sub-band output coefficients from the second set;and reconstructing a deblurred image corresponding to the first image byupsampling and filtering the set of third sub-band output coefficientsin a reconstruction portion of the filter bank.
 15. The method of claim14, further including combining the deblurred image with the secondimage to generate a composite color image.
 16. The method of claim 14,wherein the edge information is determined using an ant colonizationoptimization scheme.
 17. The method of claim 16, further includingpreparing a binary edge map in the contourlet domain and using thebinary edge map in the preparing step to select corresponding secondsub-band output coefficients from the second set.
 18. The method ofclaim 17, further including preparing a binary edge map in thecontourlet domain, dilating the binary edge map, and using the dilatedbinary edge map in the preparing step to select corresponding secondsub-band output coefficients from the second set.
 19. A method fordeblurring a blurred first image corresponding to a first color channelof a camera that also produces a sharp second image corresponding to asecond color channel of the camera, wherein the first and the secondimages each include a plurality of pixels with each pixel having anassociated respective value, the method comprising: decomposing thefirst image by filtering and downsampling to generate a first set of oneor more first sub-band output coefficients, wherein each first sub-bandoutput coefficient corresponds to a respective sub-band in a selectedone of a wavelet and a contourlet domain; decomposing the second imageby filtering and downsampling to generate a second set of secondsub-band output coefficients, wherein each second sub-band outputcoefficient corresponds to a respective sub-band in the selected domain;selecting those second sub-band output coefficients that represent edgeinformation, each of the selected second sub-band output coefficientscorresponding to a respective selected sub-band, with the selectedsub-bands together defining an edge sub-band set; preparing a third setof sub-band output coefficients which includes the selected secondsub-band output coefficients representing edge information and at leastone first sub-band output coefficient corresponding to a sub-band otherthan those sub-bands in the edge sub-band set; and reconstructing adeblurred first image by upsampling and filtering using the third set ofsub-band output coefficients as input, wherein each decomposing stepoccurs over two or more levels and the reconstructing step occurs overthe same number of levels, and wherein the number of levels ofdecomposition is determined based at least in part on a frequencyresponse of a blurring lens of the camera.
 20. A method for deblurring ablurred first image, wherein the blurred first image corresponds to afirst color channel of a camera and a second image corresponds to asecond color channel of the camera, wherein the first and the secondimages each comprise a plurality of pixels with each pixel having anassociated respective value, the method comprising: decomposing thefirst image by filtering and downsampling in a decomposition portion ofa filter bank to generate for a predetermined number of sub-bands in acontourlet domain a first set of first sub-band output coefficients,wherein each first sub-band output coefficient corresponds to arespective one of the sub-bands; decomposing the second image byfiltering and downsampling in the decomposition portion of the filterbank to generate for the predetermined number of sub-bands a second setof second sub-band output coefficients, wherein each second sub-bandoutput coefficient corresponds to a respective one of the sub-bands inthe contourlet domain; determining which of the second sub-band outputcoefficients and corresponding respective sub-bands represent edgeinformation, wherein the edge information is determined using an antcolonization optimization scheme; preparing a set of third sub-bandoutput coefficients by modifying the first set of first sub-band outputcoefficients to replace each of those first sub-band output coefficientsin the first set which correspond to the respective determined sub-bandsrepresenting edge information with the corresponding second sub-bandoutput coefficients from the second set; and reconstructing a deblurredimage corresponding to the first image by upsampling and filtering theset of third sub-band output coefficients in a reconstruction portion ofthe filter bank, wherein the method further includes: preparing a binaryedge map in the contourlet domain and using the binary edge map in thepreparing step to select corresponding second sub-band outputcoefficients from the second set; and preparing a binary edge map in thecontourlet domain, dilating the binary edge map, and using the dilatedbinary edge map in the preparing step to select corresponding secondsub-band output coefficients from the second set.